Understanding Difficulty-Based Sample Weighting with a Universal Difficulty Measure

Sample weighting is widely used in deep learning. A large number of methods essentially utilize the learning difficulty training samples to calculate their weights. In this study, scheme called difficulty-based weighting. Two important issues arise when explaining scheme. First, a unified measure that can be theoretically guaranteed for does not exist. The difficulties are determined by multiple factors including noise level, imbalance degree, margin, and uncertainty. Nevertheless, existing measures only consider single factor or part, but entirety. Second, comprehensive theoretical explanation lacking with respect demonstrating why schemes effective we prove generalization error sample as universal measure. Furthermore, provide formal justifications on role learning, consequently revealing its positive influences both optimization dynamics performance models, which instructive schemes.